Method and system for real time outlier detection and product re-binning

ABSTRACT

A method for identifying outlier devices during testing, includes: establishing binning limits for a device being tested based on one or more rules generated from external test results data of tests involving similar devices; receiving test results data in real time for the device being tested while the device is on a device tester; applying the one or more rules to the test results data for the device in real time; determining in real time, based on results of applying the one or more rules to the test results data, whether the device is an outlier with respect to the binning limits; and in response to determining that the device is an outlier, binning the outlier device separately from tested devices having test results data falling within the binning limits.

CROSS-REFERENCE TO RELATED APPLICATIONS

This applications claims the benefit of U.S. Provisional PatentApplication No. 62/760,327, filed Nov. 13, 2018, entitled “METHOD ANDSYSTEM FOR REAL TIME OUTLIER DETECTION AND PRODUCT RE-BINNING”, thedisclosure of which is hereby incorporated herein by reference in itsentirety.

U.S. Pat. No. 9,885,751, issued Feb. 6, 2018, U.S. Pat. No. 8,872,538,issued Oct. 28, 2014, U.S. Pat. No. 8,421,494, issued Apr. 16, 2013,U.S. Pat. No. 7,969,174, issued Jun. 28, 2011, U.S. Pat. No. 7,567,947,issued Jul. 28, 2009, U.S. Pat. No. 7,528,622, issued May 5, 2009, andU.S. application Ser. No. 12/497,789, filed on Jul. 6, 2009, are herebyincorporated herein by reference in their entireties for all purposes.

SUMMARY OF THE INVENTION

Methods and systems for real time outlier detection and productre-binning are provided.

According to various aspects there is provided a method for identifyingoutlier devices during testing. In some aspects, the method may include:establishing binning limits for a device being tested based on one ormore rules generated from external test results data of tests involvingsimilar devices; receiving test results data in real time for the devicebeing tested while the device is on a device tester; applying the one ormore rules to the test results data for the device in real time;determining in real time, based on results of applying the one or morerules to the test results data, whether the device is an outlier withrespect to the binning limits; and in response to determining that thedevice is an outlier, binning the outlier device separately from testeddevices having test results data falling within the binning limits.

According to various aspects there is provided a system. In someaspects, the system may include: a central rule engine configured togenerate one or more rules for taking actions with respect to testing ofa device based on external test results data of tests involving similardevices; a local rule engine configured to receive the one or morerules; and a station controller configured to receive test results datain real time for a device being tested and transmit the test resultsdata to the local rule engine in real time. The one or more rulescomprise binning limits for the device being tested. The local ruleengine may be configured to apply the one or more rules to the testresults data for the device in real time, and determine in real time,based on results of applying the one or more rules to the test resultsdata, whether the device is an outlier with respect to the binninglimits. In response to determining that the device is an outlier, thelocal rule engine may bin the outlier device separately from testeddevices having test results data falling within the binning limits.

According to various aspects there is provided a non-transitory computerreadable medium. The non-transitory computer readable medium may includeinstructions for causing one or more processors to execute operations toperform a method for identifying outlier devices during testing. In someaspects, the operations may include: establishing binning limits for adevice being tested based on one or more rules generated from externaltest results data of tests involving similar devices; receiving testresults data in real time for the device being tested while the deviceis on a device tester; applying the one or more rules to the testresults data for the device in real time; determining in real time,based on results of applying the one or more rules to the test resultsdata, whether the device is an outlier with respect to the binninglimits; and in response to determining that the device is an outlier,binning the outlier device separately from tested devices having testresults data falling within the binning limits.

According to various aspects there is provided a system for analyzingdevice test data. In some aspects, the system may include: a first ruleengine configured to generate a first rule based on test results from adevice being tested on a tester at a testing facility and a second rule,and to define and initiate a first action while the device is on thetester based on applying the first rule to the test results of thedevice; a second rule engine configured to generate the second rulebased on test results of a first plurality of other devices of the sametype tested at the testing facility and a third rule, and to define andinitiate a second action when the device is no longer on the testerbased on applying the second rule to the test results of the device; anda third rule engine configured to generate the third rule based onmanufacturing data of a second plurality of other devices of the sametype from a plurality of manufacturing facilities when the device is nolonger on the tester.

Numerous benefits are achieved by way of the various embodiments overconventional techniques. For example, the various embodiments provide acentral processor for generation of rules for real time evaluation ofdevice test results data based on a pool of external test results dataunavailable to local processors. In some embodiments, the external testresults data used to generate the rules may include test results datafor similar devices at device level, subassembly level, product level,etc., as well as failure analysis test results data. These and otherembodiments along with many of its advantages and features are describedin more detail in conjunction with the text below and attached figures.

Other features and advantages should be apparent from the followingdescription which illustrates by way of example aspects of the variousteachings of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects and features of the various embodiments will be more apparent bydescribing examples with reference to the accompanying drawings, inwhich:

FIG. 1 is a simplified block diagram of a system for outlier detectionand product re-binning in real-time according to various aspects of thepresent disclosure;

FIG. 2 is a more detailed block diagram of the system for outlierdetection and product re-binning in real-time according to variousaspects of the present disclosure;

FIG. 3 is a plot illustrating upper and lower static process limits(SPL) provided by a rule generated by the core analytics rule engineaccording to various aspects of the present disclosure;

FIG. 4 is a plot illustrating dynamic upper and lower SPL provided by arule generated in real time by the device analytics rule engineaccording to various aspects of the present disclosure; and

FIG. 5 is a flowchart of a method 500 for outlier detection and productre-binning in real-time according to various aspects of the presentdisclosure.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

While certain embodiments are described, these embodiments are presentedby way of example only, and are not intended to limit the scope ofprotection. The apparatuses, methods, and systems described herein maybe embodied in a variety of other forms. Furthermore, various omissions,substitutions, and changes in the form of the example methods andsystems described herein may be made without departing from the scope ofprotection.

Test operations involved in the manufacturing of electronic devices arecommonly performed using Automated Test Equipment (ATE), often inconjunction with and under limited control of a test station hostcomputer. After device testing has been performed, a handler places thedevices into physically separate containers referred to as “bins” underdirection of a control program, and according to the test results ofeach individual device. Binning signals that are passed from the ATE tothe handler instruct the handler to physically place the devices in theindicated bins. Typically, devices are sorted according to whether theypassed or failed testing, and further may be classified and sortedaccording to passing and failing sub-categories. Test limits may be, forexample, but not limited to, Test Program (SPEC) limits, Static ProcessLimits (SPL) and Part Average Test (PAT) limits.

Devices that pass testing may be classified and sorted based on deviceperformance during testing according to varying test result limits. Theclassifications based on test result limits are typically predeterminedand programmed into the ATE. In some cases, the classifications may bemodified, either manually, for example by a test engineer, orautomatically via feedback from the ATE from test results data obtainedat the test facility. However, a manufacturer may produce devices of thesame type at multiple locations across a country or even across theworld. Test results data for any individual manufacturing location,while typically available at a central “headquarters” location, is notavailable to other manufacturing locations. As a result, devicesmanufactured at one location can be classified and sorted differentlythat the same devices manufactured at another location. Thus, the sametype devices manufactured at different locations can have inconsistentcharacteristics for devices of the same classification.

Further, devices may be incorporated into higher level assemblies andthe higher level assemblies may be built into products. Testing may takeplace at each level of assembly and test data may be collected. The testdata from the various levels of testing may be correlated with specificdevices and may be available to the central “headquarters” location.

FIG. 1 is a simplified block diagram of a system 100 for outlierdetection and product re-binning in real-time according to variousembodiments. As used herein, the term “real time” means during a timeperiod that testing is being performed. As illustrated in FIG. 1, thesystem may include a core analytics rule engine 110, an edge analyticsrule engine 120, a device analytics rule engine 130, a test stationcontroller 140, automated test equipment (ATE) 150, and device handlingequipment 160.

Each of the core analytics rule engine 110, edge analytics rule engine120, device analytics rule engine 130, and test station controller 140may be implemented by a processor, for example but not limited to amicroprocessor, microcomputer, CPU, microcontroller, etc. One ofordinary skill in the art will appreciate that while the edge analyticsrule engine 120 and the device analytics rule engine 130 are illustratedas separate blocks, their functions may be performed by separatecomponents or may be performed by a single component without departingfrom the scope of the present disclosure. Each of the core analyticsrule engine 110, edge analytics rule engine 120, and device analyticsrule engine 130 may be connected via a network, for example, theInternet or another network, and may be configured to communicate overthe network.

The core analytics rule engine 110 may be centrally located, for exampleat a company headquarters, and may be connected via a network, forexample, the Internet or another network, to co-located facilities aswell as facilities at different geographic locations. The core analyticsrule engine 110 may receive test results data from a plurality of edgeanalytic rule engines from facilities at different geographic locations(i.e., edge facilities) as well as from a plurality of other sources,for example, but not limited to, third party test facilities, productmanufacturers (including production test and failure analysis data),etc. The core analytics rule engine 110 may have access to a pluralityof external data from multiple facilities and/or third party sourcesthat is not typically available to any particular edge analytic ruleengine. Thus, the core analytics rule engine 110 may create rules andstatic process limits (SPL) that are universally useful and current. Therules may be, for example, but not limited to, algorithms that specifyactions to be taken based on evaluation of data input to the algorithm.

Data received for the core analytics rule engine 110 may be stored in adatabase 114 that may be co-located, for example at the companyheadquarters, with the core analytics rule engine 110. Alternatively,the database 114 may be situated at a location other than at thelocation of the core analytics rule engine 110 and may be accessible viaa network, for example, the Internet or another network, by the coreanalytics rule engine 110.

The SPL may be calculated periodically (off-line) by the core analyticsrule engine 110 based on historical data. As used herein, the term“off-line” indicates that a rule engine or other device is available toperform operations while not in active communication with other ruleengines or other devices. The SPL calculated by the core analytics ruleengine 110 may be published to the device analytics rule engine 130 andapplied by the device analytics rule engine 130 to each new lot ofdevices tested in addition to the applied ATE test program limits.

The SPL may be calculated by the core analytics rule engine 110 in thecentral location based on combined external data available from theplurality of testing and manufacturing facilities. The core analyticalrule engine 110 may consider test data from a same product in otherfacilities, test data from previous test operations, fabrication data,etc., for calculating the SPL. For instance, the core analytics ruleengine 110 may use test data from a previous test operation thatmeasures a similar test parameter and set SPL to limit amount of allowedshift from a previous test operation.

In another instance, the core analytics rule engine 110 may use adistribution of a same test parameter as seen historically acrossmultiple testing sites and set SPL to limit drift of the test parameteraway from the combined distribution. In another instance, the coreanalytics rule engine 110 may use historical reliability data andcurrent manufacturing data from manufacturing execution systems (MES)for the product and tighten the SPL threshold for units (e.g., lots) ofproduct supplied from a manufacturing facility that historicallyproduced products with higher reliability risk. One of ordinary skill inthe art will appreciate that these are only examples of data that may beused and that other data collected from other external sources may beincluded without departing from the scope of the present disclosure.

The SPL may be calculated by the core analytics rule engine 110 on afixed schedule (e.g., weekly, etc.), or may be recalculated upon demandif a change in SPL is anticipated. The core analytics rule engine 110receives the external data for calculating the SPL from database 114.The SPL for each product may be published to each testing facility(i.e., edge facility) from the core analytics rule engine 110 at thecentral facility periodically (weekly, etc.) or upon demand so that theSPL are ready to be published to ATE during lot introduction.

In addition to calculating SPL, the core analytics rule engine 110 mayalso define or update rules (i.e., first rules). The first rules may bedefined (created and validated) off-line based on historical data fromthe external sources available to the core analytics rule engine 110.The first rules may be activated by the station controller 140 andapplied by the device analytics rule engine 130 in real-time to each newlot tested. The first rules may be active within a scope of a run. Thefirst rules may be continuously applied in real-time by the deviceanalytics rule engine 130 to test results data received from ATE to makedecisions and send actions accordingly (on-line). As used herein, theterm “on-line” indicates that a rule engine or other device is availableperform operations involving active communications with other ruleengines or devices. Examples of first rules defined by the coreanalytics rule engine 110 may include:

-   1. Adaptive Test Time Reduction (TTR)—adaptively augments a test    routine to skip tests to reduce test time (e.g., abort on slow    test). A threshold to abort may be determined based on historical    test time, test time from other similar ATE testing the same product    in the same test facility or in other test facilities. Examples of    systems and methods for adaptive test time reduction may be found in    U.S. Pat. Nos. 7,528,622, 7,969,174, 8,421,494, and 8,872,538    incorporated herein by reference.-   2. Bin Monitor—monitors distribution of bins and alert for outliers,    for example, alert if too many units are failing Bin 8. A threshold    may be determined based on historical quantities of units binned to    Bin 8.-   3. Freeze—detects that ATE is “frozen”—generates a fixed test data,    e.g., monitor variability (i.e., sigma) of a test that historically    has high variation.-   4. Parametric Process Capability—calculates and monitors process and    metrology capability measures (Cpk, Gage Repeatability and    Reproducibility (R&R)).-   5. Parametric Trend—detects shifts and drifts in test results, e.g.,    compare current test result to test results from previous test    operations.-   6. Site-to-Site Deviations—detects unexpected differences in test    results between sites during parallel testing.-   7. Tester Settings—validates that ATE settings and configuration are    correct, for example, compare current operation with operation in    cases where ATE is under control of an entity different from second    rules provided by the core analytics rule engine 110.-   8. Machine Learning based System Level Test (SLT) reduction—predicts    SLT outcome and skips SLT on units that are unlikely to fail. The    core analytics rule engine 110 may apply machine learning to    historical SLT and/or Burn-In outcomes and test data from preceding    test operations to build a model predicting an outcome of Burn-In or    of SLT based on the test data. A rule may be defined based on the    model for deciding whether a new unit being tested at the test    operation is to skip SLT and/or Burn-In based on its test data.    One of ordinary skill in the art will appreciate that the listed    first rules are exemplary and that other rules may be defined    without departing from the scope of the present disclosure.

The core analytics rule engine 110 may generate the first rules andvalidate them (i.e., ensure that rule outcome is consistent with theexpectations) based on the external test data available in the database114. The core analytics rule engine 110 may update the first rules basedon newly available external data in order to keep them current. Anexample rule management system is described in U.S. Pat. No. 7,567,947incorporated herein by reference. The core analytics rule engine 110 atthe central facility may publish the first rules for each product to theedge analytics rule engines 120 at each testing facility on a periodicbasis (e.g., weekly, etc.) or upon demand so the first rules are readyto be activated during lot introduction and applied during a run.

The edge analytics rule engine 120 may be similar to the core analyticsrule engine 110 with the exception that the edge analytics rule engine120 only has access to data from the facility in which it is located andtherefore any rules (i.e., second rules) and SPL generated by the edgeanalytics rule engine 120 are more limited in scope. The second rulesmay be generated by the edge analytics rule engine 120 at each locationwhere devices are tested from test results data and device binning datafor device testing performed at the geographic location. The SPL and thesecond rules generated by the edge analytics rule engine 120 aretypically product specific, i.e., the rules and SPL may vary fromproduct to product. Upon lot introduction to the ATE, the stationcontroller 140 receives information identifying the product, e.g.,product name, facility, test program, etc., selects applicable SPL andactivates applicable second rules based on this information.

The device analytics rule engine 130 may receive real time test datafrom the station controller 140, apply SPL and/or first or second rulesidentified by the station controller 140 to the test data, and make adecision that is communicated to the ATE via the station controller 140.For example, the device analytics rule engine 130 may “overwrite” thebinning that was decided by the ATE program with its own final binningassignment, decide to abort the testing, issue an alert, augment thetest sequence. An example binning system is described in U.S. patentapplication Ser. No. 12/497,798, and an example system for modifyingexecution of a test sequence is described in U.S. Pat. No. 9,885,751,both of which are incorporated herein by reference. In accordance withvarious aspects of the present disclosure, the device analytics ruleengine 130 may be implemented as part of the station controller 140.

The device analytics rule engine 130 may define a third rule. The thirdrule may be defined by modifying a first or second rule obtained fromthe core analytics rule engine 110 or the edge analytics rule engine120, respectively, by creating a new rule from scratch, or by combiningmultiple rules and/or SPLs. For instance, if the first rule obtainedfrom the core analytics rule engine 110 is a PAT rule that specifies a“local” baseline to calculate the test limits, the device analytics ruleengine 130 may calculate the baseline based on test data it collected sofar on the current lot. The device analytics rule engine 130 may definea new third rule with the limits based on the “local” baseline. Thethird rule may specify how PAT limits are to be calculated orrecalculated, for example, a number of units to consider for a baseline,a time period to consider, how far from baseline to set the limits, etc.In some cases, the device analytics rule engine 130 may in real timecombine multiple sets of limits. For example a new third rule may bedefined to apply a set of limits which is a logical “AND” of the SPEClimits, the SPL from the station controller 140, and the PAT limits itcalculated, to determine a final binning assignment

The device analytics rule engine 130 may receive real time device testresults data from the station controller 140, apply first, second, orthird rules to the test results data in real-time to generate one ormore instructions to modify device binning in real time before thedevice under test is removed from the ATE. The device analytics ruleengine 130 may transmit the instructions to the test station controller140. Based on the instructions received from the device analytics ruleengine 130, the test station controller 140 may provide modified binninginstructions to the device handling equipment 160 for binning of devicesthat pass ATE test.

The ATE 150 may perform functional testing of devices and provide testresults data to the test station controller 140. The test stationcontroller 140 may provide binning instructions to the device handlingequipment 160. The device analytics rule engine 130 may modify thedevice binning instructions by applying in real-time rules received fromthe edge analytics rule engine 120, the core analytics rule engine 110,or rules generated by the device analytics rule engine 130 itself to thetest results data.

FIG. 2 is a more detailed block diagram of the system 100 for outlierdetection and product re-binning in real-time according to variousembodiments. Binning of tested devices may be performed based on limitsprovided by rules generated by the core analytics rule engine 110 usingexternal data received from a plurality of sources. The core analyticsrule engine 110 may be centrally located, for example but not limitedto, at a headquarters location of a company. Device test results datamay be collected at the headquarters location from multiple externalsites, for example but not limited to device test sites, devicemanufacturing facilities, etc.

The core analytics rule engine 110 may receive the external test resultsdata collected from the multiple sites and/or multiple levels of test,for example, but not limited to, third party test facilities, productmanufacturers (including production test and failure analysis data),etc., for a particular device type. The external test results data mayinclude, for example but not limited to, upper and lower static processlimits (SPL) for binning of a particular device type at thecorresponding locations. One of ordinary skill in the art willappreciate that other types of test results data may be included in theexternal test results data without departing from the scope of thepresent disclosure. Data received for the core analytics rule engine 110may be stored in a database 114 that may be co-located with the coreanalytics rule engine 110 or may be situated at a location other than atthe location of the core analytics rule engine 110.

The core analytics rule engine 110 may analyze the received externaltest results data generate one or more first rules 210 based on thereceived external test results data. The one or more first rules mayinclude static process limits (SPL). The core analytics rule engine 110may have access to a plurality of external data from multiple facilitiesand/or third party sources that is not typically available to anyparticular edge analytic rule engine. For example, the core analyticsrule engine 110 may generate a first rule 210 based on the receivedexternal test results data that applies upper and lower SPL for binningof tested devices.

The core analytics rule engine 110 may publish the one or more firstrules 210 to the device analytic rule engines 130 and/or the edgeanalytics rule engines 120 for each of the plurality of facilities atdifferent geographic locations. The core analytics rule engine 110 maypublish the one or more first rules 210 on a periodic basis, for exampleonce per week or another time period, or may publish the one or morefirst rules 210 on demand. At each of the different geographiclocations, when a new lot of the particular device type is tested by theautomated test equipment (ATE) 150, the device analytic rule engines 130and/or the edge analytics rule engines 120 may select an appropriatefirst rule 210 (e.g., a rule related to a particular device type beingtested) and transmit the first rule 210 to the test station controller140 configured to accept the input from the device analytic rule engines130 and/or the edge analytics rule engines 120.

The test station controller 140 may receive the test data results ofeach sequential device test and transmit the test data results of thedevice to the device analytic rule engine 130 in real-time. The deviceanalytic rule engine 130 may apply the first rule 210 generated based onexternal test results data to the test data results for the device inreal-time before the device leaves the ATE 150. For example, applicationof the first rule 210 to the test data results for the device may causethe device to be binned according to the first rule 210 rather thanaccording to the binning instructions 234 from the ATE 150. The teststation controller 140 may cause binning instructions to be sent to thedevice handling equipment 160 to sort devices according to the firstrule 210. Thus, a device may pass testing by the ATE 150 but may notcomply with the first rule, i.e., may be an outlier, and will thereforebe binned differently than the devices that pass test and comply withthe first rule.

The test station controller 140 may also periodically transmit thedevice test data results to the edge analytics rule engine 120. The edgeanalytics rule engine 120 may analyze the device test data results togenerate a second rule that may be more appropriate for the testingconducted at the associated facility. The edge analytics rule engine 120only has access to data from the facility in which it is located andtherefore any rules (i.e., second rules) and SPL generated by the edgeanalytics rule engine 120 are more limited in scope. The edge analyticsrule engine 120 may periodically transmit device test results data 221to the core analytics rule engine 110 (and the database 114). Thus, thecore analytics rule engine 110 may update and transmit the first rule210 to the multiple sites periodically or on demand.

In accordance with various aspects of the present disclosure, the deviceanalytics rule engine 130 may define a third rule. The third rule may bedefined by modifying a first or second rule obtained from the coreanalytics rule engine 110 or the edge analytics rule engine 120,respectively, by creating a new rule from scratch, or by combiningmultiple rules and/or SPLs.

FIG. 3 is a plot 300 illustrating upper and lower SPL provided by a rulegenerated by the core analytics rule engine 110 in accordance withvarious aspects of the present disclosure. As illustrated in FIG. 3, thecore analytics rule engine 110 may publish the first rule 210 to theedge analytics rule engine 120 and/or the device analytics rule engine130, and the edge analytics rule engine 120 and/or the device analyticsrule engine 130 may transmit the first rule 210 to the test stationcontroller 140. The first rule 210 may include upper and lower SPL 310,312 for binning devices that pass ATE 150 testing within the specifiedupper and lower test limits 320, 322. FIG. 3 shows that the upper andlower SPL 310, 312 for binning devices specified by the first rule 210are narrower than the specified upper and lower test limits 320, 322from the ATE 150.

The test results data 350 show a first outlier 355 and a second outlier357. Although the first outlier 355 and a second outlier 357 in the testdata results 350 correspond to devices that pass ATE 150 testing, theyare outliers with respect to the upper and lower SPL 310, 312 forbinning devices specified by the first rule 210. Therefore, the teststation controller 140 may provide binning instructions to the devicehandling equipment 160 to bin the devices corresponding to the first andsecond outliers 355, 357 differently than the devices corresponding tothe test results 350 that fall within the upper and lower SPL 310, 312for binning devices specified by the first rule 210 even though thedevices corresponding to the first and second outliers 355, 357 passedATE 150 testing. This re-binning of the devices can occur in real timesuch that a new bin may be assigned for a device before the devicefinishes its respective ATE testing.

In accordance with various aspects of the present disclosure, adaptivetesting may be performed on real time data from the ATE 150 to adjustthe rules generated by the rule engines to modify the binninginstructions. Referring to again to FIG. 2, the first rule (i.e., SPLfrom the core analytics rule engine) 210 based on external test resultsdata may be published to the device analytics rule engine 130 and/or theedge analytics rule engine 120 by the core analytics rule engine 110.The device analytics rule engine 130 and/or the edge analytics ruleengine 120 may initially transmit the first rule 210 to the test stationcontroller 140. Real time test results data 230 from the ATE 150 may bereceived by the test station controller 140 and transmitted to thedevice analytics rule engine 130.

The device analytics rule engine 130 may calculate values, for examplebut not limited to a part average test limit (PAT), in real time asdevices are being tested, generate a new rule (i.e., the third rule) 220based on the real time test results data, and transmit the third rule220 to the test station controller 140. For example, the third rule 220may provide narrower upper and lower binning limits than the SPLprovided by the first rule. The device analytics rule engine 130 maycontinue to modify the third rule 220 in real time as devices are testedsuch that the modified third rule may cause re-binning the devicecurrently under ATE 150 testing before it is removed from the ATE 150.The real time upper and lower binning limits specified by the third rule220 may be modified after sufficient real time device test results datawithin a device test lot are received by the test station controller 140and transmitted to the device analytics rule engine 130. Alternatively,the real time upper and lower binning limits specified by the third rule220 may be modified and applied on a device.

FIG. 4 is a plot 400 illustrating upper and lower SPL provided by a rulegenerated in real time by the device analytics rule engine 130 inaccordance with various aspects of the present disclosure. Referring toFIG. 4, the same test data results 350 are illustrated with the sameupper and lower test limits 320, 322 and upper and lower SPL 310, 312for binning devices as illustrated in FIG. 3. As further illustrated inFIG. 4, real time upper and lower binning limits 410, 412 are providedby the third rule 220 modified in real time by the device analytics ruleengine 130 based on the real time test results data 230 from the ATE150. The real time upper and lower binning limits 410, 412 may benarrower than the upper and lower SPL 310, 312 specified by the firstrule 210 for binning devices.

FIG. 4 again shows the first outlier 355 and the second outlier 357based on the upper and lower SPL 310, 312 for binning devices specifiedby the first rule 210, and also shows an additional third outlier 455detected based on the real time upper and lower binning limits 410, 412provided by the real time modification of the third rule 220 by thedevice analytics rule engine 130. The upper and lower SPL 310, 312 forbinning devices specified by the first rule 210 may be applied untilsufficient real time device test results data within a device test lotare collected by the test station controller 140 and transmitted to thedevice analytics rule engine 130 to calculate values to modify the thirdrule 220 to provide the real time upper and lower binning limits 410,412. A number of test points define “sufficient real time device testresults data” may be specified for the device analytics rule engine 130based on a desired statistical confidence and quantities of productunits being tested or by another method. Alternatively, the real timeupper and lower binning limits 410, 412 may be modified and applied on adevice by device basis as each device is tested.

The test station controller 140 may cause binning instructions to besent to the device handling equipment 160 to bin devices that complywith the real time modified third rule 220 differently than devices thatdo not comply with the real time modified third rule 220 (i.e.,outliers). Thus, additional outlier devices may be detected withintighter limits over time and/or in real time. The currently appliedbinning rules may be monitored via a user interface at a headquarterslocation and/or at test locations.

Systems and methods provided in accordance with various aspects of thepresent disclosure may provide benefits during new product introduction(NPI). During NPI, little or no statistical test information isavailable for the new product. Conventional systems require many devicesto be tested in order to collect and analyze device test results todetermine binning limits.

In accordance with various aspects of the present disclosure, the deviceanalytics rule engine 130 may establish rules in real-time while devicesare still on the ATE 150. The real time rules established by the deviceanalytics rule engine 130 may quickly define upper and lower binninglimits and identify outlier devices.

FIG. 5 is a flowchart of a method 500 for outlier detection and productre-binning in real-time in accordance with various aspects of thepresent disclosure. Referring to FIG. 5, at block 510 the core analyticsrule engine 110 may receive a plurality of device test data results froma plurality of sources, for example, but not limited to, edge analyticrule engines, third party test facilities, product manufacturers(including production test and failure analysis data), etc., for aparticular device type tested in a plurality of different locations andlevels of test. At block 515, the core analytics rule engine 110 maygenerate a first rule 210. For example, the core analytics rule engine110 may generate one or more first rules 210 based on the external testresults data available to the core analytics rule engine 110.

At block 520, the core analytics rule engine 110 may publish the firstrule 210 to the device analytic rule engine 130 and/or the edgeanalytics rule engine 120. The core analytics rule engine 110 maypublish the first rule 210 on a periodic basis, for example once perweek or another time period. Alternatively, the core analytics ruleengine 110 may publish the first rule 210 in real time as additionaldevice test data results are received from test facilities and othersources, for example, but not limited to, third party test facilities,product manufacturers (including production test and failure analysisdata), etc., at different geographic locations.

At block 525, the device analytic rule engine 130 and/or the edgeanalytics rule engine 120 may transmit the first rule to the teststation controller 140. For example, when a new lot of the particulardevice type is tested by the automated test equipment (ATE) 150, thefirst rule 210 may be transmitted to the test station controller 140.The first rule 210 may specify upper and lower SPL for binning devicesthat pass ATE 150 testing. At block 530, the ATE 150 may perform adevice test.

At block 535, the device analytics rule engine 130 may receive real timetest results data for the tested device. For example, real time testresults data 230 from the ATE 150 may be received by the test stationcontroller 140 and transmitted to the device analytics rule engine 130.

At block 540, the device analytics rule engine 130 may apply the firstrule to the test results data in real time. The device analytics ruleengine 130 may calculate values, for example but not limited to a partaverage test limit (PAT), in real time as devices are being tested. Thefirst rule may provide narrower upper and lower binning limits than theSPL provided by the ATE program. The device analytics rule engine 130may modify the first rule and forward the modified rule to the teststation controller 140 in real time based on continually received realtime test results data.

At block 545, device analytics rule engine 130 may identify devices thatdo not fall within the upper and lower binning limits (i.e., outliers)established by the first rule (see FIG. 4). In accordance with variousaspects of the present disclosure, the device analytics rule engine 130may apply the upper and lower SPL for binning devices specified by thefirst rule 210 until sufficient real time test results data is collectedto enable the device analytics rule engine 130 to generate a new rule.

At block 550, the test station controller 140 may cause binninginstructions from the device analytics rule engine 130 to be sent to thedevice handling equipment 160 to bin devices that comply with the realtime modified third rule 220 differently than devices that do not complywith the first rule (i.e., outliers).

At block 555, the test station controller 140 may periodically transmittest results data to the edge analytics rule engine 120. The edgeanalytics rule engine 120 may also generate new rules. However, the edgeanalytics rule engine 120 only has access to data from the facility inwhich it is located and therefore any rules (i.e., second rules) and SPLgenerated by the edge analytics rule engine 120 are more limited inscope. At block 560, the edge analytics rule engine 120 may transmit thetest results data to the core analytics rule engine 110. Data receivedfor the core analytics rule engine 110 may be stored in a database 114.At block 565, the core analytics rule engine 110 may update the firstrule based on the new test results data from the edge analytics ruleengine 120 as well as from other external sources.

The method 500 may be embodied on a non-transitory computer readablemedium, for example, but not limited to, a memory, storage device, orother non-transitory computer readable medium known to those of skill inthe art, having stored therein a program including computer executableinstructions for making a processor, computer, or other programmabledevice execute the operations of the methods, and may be distributedamong the various rule engines and station controllers and associatedstorage devices as described in accordance with various aspects of thepresent disclosure.

One of ordinary skill in the art will appreciate that other aspects maybe contemplated, for example but not limited to establishing binningrules having multiple process limits defining various performancecharacteristic (e.g., clock speed, etc.) for the devices that pass ATEtesting without departing from the scope of the present disclosure.

The examples and embodiments described herein are for illustrativepurposes only. Various modifications or changes in light thereof will beapparent to persons skilled in the art. These are to be included withinthe spirit and purview of this application, and the scope of theappended claims, which follow.

What is claimed is:
 1. A method for identifying outlier devices duringtesting, the method comprising: establishing binning limits for a devicebeing tested based on one or more rules generated from external testresults data of tests involving similar devices; receiving test resultsdata in real time for the device being tested while the device is on adevice tester; applying the one or more rules to the test results datafor the device in real time; determining in real time, based on resultsof applying the one or more rules to the test results data, whether thedevice is an outlier with respect to the binning limits; and in responseto determining that the device is an outlier, binning the outlier deviceseparately from tested devices having test results data falling withinthe binning limits.
 2. The method of claim 1, wherein the external testresults data comprise at least one of device test results data generatedat a facility different from a facility at which the device is beingtested, higher level assembly test results data, product test resultsdata, and failure analysis test results data.
 3. The method of claim 1,wherein the one or more rules are generated at a facility having accessto the external test results data, and wherein the facility havingaccess to the external test results data is different from a facility atwhich the device is being tested.
 4. The method of claim 1, wherein theone or more rules comprise instructions to add or omit tests for thedevice, or static process limits (SPL) applied to the test results datafor the device subsequent to SPL applied by the device tester.
 5. Asystem for identifying outlier devices during testing, the systemcomprising: a central rule engine configured to generate one or morerules for taking actions with respect to testing of a device based onexternal test results data of tests involving similar devices; a localrule engine configured to receive the one or more rules; and a stationcontroller configured to receive test results data in real time for adevice being tested and transmit the test results data to the local ruleengine in real time; wherein the one or more rules comprise binninglimits for the device being tested, wherein the local rule engine isconfigured to apply the one or more rules to the test results data forthe device in real time, and determine in real time, based on results ofapplying the one or more rules to the test results data, whether thedevice is an outlier with respect to the binning limits; and in responseto determining that the device is an outlier, binning the outlier deviceseparately from tested devices having test results data falling withinthe binning limits.
 6. The system of claim 5, wherein the external testresults data comprise at least one of device test results data generatedat a facility different from a facility at which the device is beingtested, higher level assembly test results data, product test resultsdata, and failure analysis test results data.
 7. The system of claim 5,wherein the central rule engine is located at a facility having accessto the external test results data, and wherein the facility havingaccess to the external test results data is different from a facility atwhich the device is being tested.
 8. The system of claim 5, wherein theone or more rules comprise instructions to add or omit tests for thedevice, or static process limits (SPL) applied to the test results datafor the device by the local rule engine subsequent to SPL applied by adevice tester.
 9. A system for analyzing device test data, the systemcomprising: a first rule engine configured to generate a first rulebased on test results from a device being tested on a tester at atesting facility and a second rule, and to define and initiate a firstaction while the device is on the tester based on applying the firstrule to the test results of the device; a second rule engine configuredto generate the second rule based on test results of a first pluralityof other devices of a same type tested at the testing facility and athird rule, and to define and initiate a second action when the deviceis no longer on the tester based on applying the second rule to the testresults of the device; and a third rule engine configured to generatethe third rule based on manufacturing data of a second plurality ofother devices of the same type from a plurality of manufacturingfacilities when the device is no longer on the tester.
 10. The system ofclaim 9, wherein the first rule engine is configured to generate thefirst rule and define and initiate the first action while the first ruleengine is online.
 11. The system of claim 9, wherein the second ruleengine is configured to generate the second rule and define and initiatethe second action while the second rule engine is offline.
 12. Thesystem of claim 9, wherein the third rule engine is configured togenerate the third rule while the third rule engine is offline.
 13. Thesystem of claim 9, wherein the manufacturing data includes test resultsfrom testing of devices of the same type.
 14. The system of claim 9,wherein the plurality of manufacturing facilities includes the testingfacility.
 15. The system of claim 9, wherein the first rule engine isdisposed at the testing facility and is configured to generate the firstrule during testing of the device.
 16. The system of claim 9, wherein:the first rule engine is operable to provide real time analytics at apredetermined edge facility, and the second rule engine is operable toprovide off-line analytics at the predetermined edge facility.
 17. Thesystem of claim 9, wherein the third rule engine is operable to provideoff-line analytics at a central facility that is connected to multipleedge facilities.
 18. The system of claim 9, wherein the second ruleengine is disposed at the testing facility and is configured to generatethe second rule based on test results of the first plurality of otherdevices of the same type tested at the testing facility obtained by thesecond rule engine prior to testing the device, and is configured tocommunicate the second rule to the to the first rule engine.
 19. Thesystem of claim 9, wherein the third rule engine is disposed at alocation different from the testing facility and is configured togenerate the third rule based on test results of the second plurality ofother devices of the same type tested at a plurality of facilitiesobtained by the third rule engine prior to testing the device, and isconfigured to communicate the third rule to the to the first ruleengine.
 20. The system of claim 9, wherein the first action is one ofrebinning of the device, detecting the device as an outlier, a test timereduction (TTR) action, and aborting test, and wherein the second actionis one of rebinning of the device, detecting the device as an outlier,shutting down the tester, performing maintenance on the tester, andmachine learning (ML)-based test time reduction (TTR) action.